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Mid-Term  Report 

PROPOSAL  NO.:  2002011035 


TITLE:  Expert  System  Utilization  for  Behavioral  Health 


ACCOMPLISHMENTS 

In  behavioral  health  (BH),  considerable  resources  are  allocated  to  the  basic,  standard 
processes  of  conducting  initial  evaluations.  Patients  present  to  BH  clinics  much  like  they 
do  to  an  emergency  room  or  sick  call;  problems  range  from  minor  to  severe,  urgent  to 
routine,  administrative  to  medical  needs.  Triaging  patients  is  required  to  determine  an 
appropriate  immediate  disposition.  The  critical  yield  of  the  evaluation  is  actuarial  data 
used  to  predict  risk.  Historically,  this  has  not  been  optimized  for  either  resource 
allocation  or  quality  of  care,  resulting  in  delays  in  service,  additional  cost  to  provide  care, 
and  inconsistent  scientific  application  of  BH. 

This  project  originally  proposed  to  apply  decision  algorithms  to  an  automated  intake 
system  to  directly  address  quality,  cost  and  access  limitations.  The  key  tasks  were: 

1.  Defining  an  explicit  knowledge  base  used  to  make  triage  and  assessment  decisions. 

2.  Providing  a  detailed  analysis  of  current  best  practices  and  evidence-based  decision¬ 
making  criteria. 

3.  Creating  symbolic  logic  using  the  identified  criteria. 

4.  Designing  output  that  is  useful  to  providers. 

5.  Systematically  evaluating  the  decisions  and  logic  against  clinician  judgment. 

6.  Integrating  the  outcome  of  the  project  with  larger  initiatives  (BHAVRS,  DCSP, 
CHCSII). 

It  was  planned  to  accomplish  this  application  of  AI  technology  to  BH  triage  by  providing 
a  detailed  analysis  of  current  best  practices  and  evidence-based  decision-making  criteria, 
and  by  systematically  presenting  the  results  to  a  panel  of  experienced  providers.  The 
providers  would  then  be  guided  by  a  knowledge  developer  to  elicit  the  decision  points 
factors  that  would  apply  in  various  clinical  scenarios.  These  findings  would  be 
incorporated  into  an  Expert  System  to  enhance  the  business  process  of  BH  triage. 
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The  first  step  to  develop  the  expert  system  was  to  define  the  knowledge  base  used  for 
making  triage  and  risk  assessment  decisions.  Therefore,  it  was  necessary  to  develop  a 
formal  model  of  the  decision  algorithms  that  are  used  in  triage  situations.  Several 
prototypes  for  the  flow  of  decision  factors  were  developed  and  subsequently  presented  to 
the  expert  panel  of  providers,  thus  beginning  of  the  process  for  eliciting  decision  points 
for  the  expert  system.  The  models  presented  were  based  upon  empirical  research,  clinical 
practice  guidelines  for  depression,  suicide,  deployment  health  and  post-traumatic  stress 
disorder.  Upon  presentation  of  the  models,  the  providers  participated  in  semi-structured 
interviews  in  order  to  elicit  quantifiable  feedback.  Feedback  was  considered  in  model 
revisions. 

Concurrent  with  model  evaluation,  the  development  of  the  Artificial  Intelligence  (AI) 
technology  was  also  initiated.  Research  in  the  field  of  expert  systems  identified  a 
knowledge  acquisition  and  automation  expert  system  shell  called  Exsys  CORVID.  This 
software  is  a  development  package  that  aids  in  applying  human  knowledge  in  a  specific 
area  of  expertise  to  make  decisions,  typically  to  solve  a  problem  or  make  a 
recommendation.  CORVID  was  tested,  and  it  satisfied  the  technological  needs  to  reach 
the  stated  goals  of  this  project,  so  it  was  acquired. 

Though  validation  of  the  decision  process  continues,  the  creation  of  symbolic  logic 
representing  the  decision  models  and  expert  input  has  begun.  CORVID  is  designed  to 
develop  a  rule-based  expert  system  that  will  ultimately  allow  symbolic  representation  of 
the  logic  and  decision  process,  and  will  support  integration  with  a  clinical  encounter  that 
is  derived  from  the  interaction  between  a  patient  and  the  computer.  CORVID  is  also 
capable  of  interfacing  with  electronic  records  and  database  technologies,  offering 
potential  for  a  greater  level  of  data  driven  decision  support,  and  further  supporting  the 
goals  of  the  project.  The  first  system  prototype  is  scheduled  for  completion  for  pilot 
testing  and  further  feedback  gathering  at  the  approximate  time  of  this  mid  term  report. 


PI  Evaluation:  Project  Accomplishments  Match  Proposal 


PROBLEMS/ISSUES 

The  project  has  encountered  some  problems/issues  as  of  the  date  of  the  report.  In  the 
original  proposal,  it  was  explained  that  an  electronic  record  would  provide  a  critical 
enabling  platform  for  an  expert  system  because  both  providers  and  patients  can  be 
expected  to  directly  input  information  into  the  record.  The  CBHRS  was  identified  as  a 
prototype  for  an  enabling  platform  for  the  decision  system  program  while  the  larger 
coordination  issues  for  the  BH  enterprise  record  were  addressed.  At  present  there  is  no 
integrated  electronic  record  system  in  use  in  Army  BH  clinics.  Continuing  efforts  in  the 
Military  Health  System  to  develop  an  integrated  electronic  record  have  produced 
multiple  prototypes.  It  seems  that  the  CBHRS  platform  may  be  at  the  point  of  being 
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phased  out  of  use,  in  favor  of  other  interim  solutions.  As  a  result,  the  exploration  of  an 
electronic  record  as  an  enabling  platform  now  requires  assessment  of  which  prototype  to 
use,  and  this  has  set  the  project  back,  due  to  the  additional  effort  this  will  require.  At  the 
present  time  we  are  exploring  the  ability  of  this  product  to  function  with  ICDB  and  the 
Fusion  effort  as  well  as  with  CHCSII. 


Pi's  Evaluation:  Project  Accomplishments  Close  to  Proposal 
Project  encountered  no  significant  problems/issues 


Second  Half  Project  LifeCycle 

It  was  identified  in  the  original  proposal  that  it  would  be  essential  for  the  project’s  success 
to  have  adequate  development  of  the  technology  and  a  phased  implementation  plan. 

Plans  for  the  second  half  of  the  project  are: 

Finish  the  initiated  development  and  definition  of  the  explicit  knowledge  base  and 
decision  process  model. 

Using  CORVID,  complete  the  creation  and  implementation  of  the  symbolic  logic  based  on 
the  elicited  expert  knowledge. 

Evaluate  the  decisions  and  logic  against  clinician  judgment. 

Design  output  that  is  useful  to  providers.  Explore  and  evaluate  possibilities  for  interfacing 
the  expert  system  with  electronic  records,  as  well  as  other  initiatives. 


Pi's  Evaluation:  Project  Plan  is  according  to  Proposal. 


Deliverable  Update 

We  originally  proposed  to  measure  two  general  categories  of  outcome  metrics  for  this 
project:  user  acceptance/satisfaction  and  “cost”  savings.  User  acceptance  of  the 
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technology  was  to  be  measured  as  demonstrated  both  by  subjective  report  and  system 
utilization.  At  this  point  in  the  project,  we  are  not  able  to  provide  data  for  either  metric, 
as  we  have  not  yet  implemented  the  deployment  phase  of  the  project.  The  metrics  are  still 
deemed  to  be  appropriate  to  the  project.  So  far,  there  has  been  a  general  acceptance  on 
the  part  of  providers  in  regards  to  the  expert  system  development,  and  there  has  been 
polite  cooperation  with  the  systems  analyst  in  the  process  of  gathering  the  expert 
knowledge.  It  is  still  a  concern  that  as  the  project  reaches  implementation  phases, 
resistance  may  be  encountered.  However,  it  is  believed  that  our  phased  implementation 
plan,  along  with  our  emphasis  on  provider  validation,  will  alleviate  any  significant 
ambivalence  that  may  arise. 


Pi's  Evaluation:  Deliverable  is  on  schedule,  per  Proposal 


Expenditures 


4Q 

3Q  FY  01  FY 


01 

Element  of 

Resource  (EOR) 

Jun  1 

Apr  1  -  May  31  -  Sep 
30 

Travel  2100 

4,488.05  0.00 

Shipping  2200 

0.00  0.00 

Rent  & 

Communications 

2200 

0.00  0.00 

Contract  for 

Services 

2500 

161,974.24  0.00 

Supplies 

2600 

20,037.71  0.00 

Equipment 

3100 

0.00  0.00 
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Financial  Narrative 


Travel  was  required  for  the  PI  to  attend  a  course  in  AI.  In  addition,  travel  was  required 
to  attend  training  in  the  software  system. 

Contracts  for  services  were  purchased  for  a  Knowledge  Developer,  a  CORVID 
programmer,  and  a  Research  Assistant. 

Supplies  in  the  form  of  software  were  purchased,  including  CORVID,  AI  Trilogy,  and 
Copernic  Agent  Pro.  CORVID  is  the  expert  system  shell,  AI  Trilogy  is  another  AI 
product  that  was  explored,  but  will  not  be  used  in  the  initial  phase  of  this  project,  and 
Copernic  Agent  Pro  is  a  tool  that  monitors  research,  and  was  purchased  to  facilitate  the 
process  of  gathering  knowledge. 


Pi’s  Evaluation:  Budget  and  Resources  in  line  with  Proposal 


*  *  *  END  OF  REPORT  *  *  * 


